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Executive Summary 

In 2003, NCES will be releasing results for both the 2001 PIRLS fourth-grade assessment 
and the 2002 NAEP fourth grade reading assessment. In andcipatdon of questions about 
how these two assessments compare, NCES convened an expert panel to compare the 
content of the PIRLS and NAEP assessments and determine if they are measuring the same 
construct. This involved a close examinadon of how PIRLS and NAEP define reading, the 
texts used as the basis for the assessments, and the reading processes required of students in 
each. 



The comparison of the NAEP and PIRLS fourth-grade reading assessments suggests that 
there is a great deal of overlap in what the two assessments are measuring. While they do 
seem to defining and measuring the same kind of ‘‘reading,” PIRLS is an easier assessment 
than NAEP, with more text-based tasks and shorter, less complex reading passages. The 
similarides and differences between the two are Listed below. 

Similarities 

• PIRLS and NAEP define “reading” similarly, as a construcdve process. 

• PIRLS an(^ NAEP assess reading for a literary experience and reading to be 
informed. 

• PIRLS and NAEP call for students to develop interpretadons, make connecdons 
across text, and evaluate aspects of what they have read. 

• PIRLS and NAEP use authendc texts as the basis for the reading assessment. 

• PIRLS and NAEP use muldple-choice and constructed response quesdons with 
similar distribudons of these types of quesdons. In both, about half of the items 
are constructed-response format. 

Differences 

• PIRLS calls for more text-based interpretadon than NAEP. NAEP places more 
emphasis on having students take what they have read and connect to other 
readings or knowledge and to cridcaUy evaluate what they have read. 

• Close to 20% (18%) of the items in PIRLS require students to locate informadon 
in the text that is virtually an idendcal match to what is in the stem of the item. 
NAEP does not have any items requiring a verbadm match. 

• PIRLS reading passages are, on average, about half the length of the NAEP 
reading passages. PIRLS passages are, on average, about 547 words, while NAEP 
passages are, on average, about 1000 words. 
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Results of readability analyses suggest that the PIRLS reading passages are easier 
than the NAEP passages (one to two grade levels lower, on average) 
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A Content Comparison of the NAEP and PIRLS 
Fourth-Grade Reading Assessments 



Introduction 

In April 2003, NCES will release results of fourth-grade students’ reading achievement in the 
United States from two assessments; the 2002 Nadonal Assessment of Educational Progress 
(NAEP) reading assessment and the 2001 Progress in International Reading Literacy Study 
(PIRLS) reading literacy assessment. PIRLS is a study of the International Association for 
the Evaluation of Educadonal Achievement (lEA) and was administered in 2001 in 35 
countries. It is andcipated that when the results for NAEP and PIRLS are released there will 
be quesdons about how the two assessments compare. Policymakers, educators, and the 
general public are likely to wonder whether NAEP and PIRLS are assessing the same thing. 
In andcipadon of this quesdon, a comparison of the content of the two fourth-grade 
assessments was carried out. The frameworks, reading passages, and assessment items for 
each assessment were examined and systemadcally compared. This paper describes the 
analysis and how the content of the two assessments compare. 

The NAEP reading assessment is based on a framework first developed for the 1992 
assessment through a widely deliberadve process involving teacher, curriculum specialists, 
reading researchers, policymakers, and the general public represendng a broad cross-secdon 
of the United States (Nadonal Assessment Governing Board (NAGB), 2001/ The PIRLS 
framework and assessment were developed through a coUaboradve process involving 
reading and assessment specialists from the pardcipadng countries (Campbell et al. 2001). 
Both NAEP and PIRLS assessed nadonally-representadve samples of fourth-graders in the 
United States. The internadonal target populadon for PIRLS was the “upper of the two 
adjacent grades with the most nine-year-olds”; this corresponds to the fourth grade in most 
countries but the third and fifth grades in some countries in the study. 

The NAEP -PIRLS comparison study reported on in this paper is based on a similar 
comparison conducted by NCES of the lEA’s 1991 reading literacy study and NAEP 
(Binkley and Rust, 1994). In the 1991 lEA reading literacy study. United States fourth- 
graders performed very well compared to their counterparts in other countries, ranking 
second out of 23 countries. This was incongruous with how fourth-grade reading 
achievement was being reported by NAEP— only 29 percent of fourth -graders in 1992 met 
the Proficient Achievement Level set by the National Assessment Governing Board (Mullis, 
Campbell, and Farstrup 1993). In the analysis of the 1991 lEA study and NAEP, the two 
assessments were compared in terms of how they defined reading, the aspects of reading 
they measured, and the kinds of texts they used. The analysis revealed that the 1991 IE A 
reading literacy study assessed a small subset of what NAEP assessed, and contained less 
challenging texts and tasks (Binkley and Williams 1996). 

The 2001 PIRLS assessment is based on a new framework that aims to assess a broader and 
higher level of reading than the 1991 study. The passages in PIRLS are longer and more 
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developed than those in the 1991 lEA study, and the kinds of quesdons that students are 
asked are more probing and cover a broader range of reading processes. A comparison of 
the 1991 lEA study and PIRLS showed that these two assessments are markedly different 
with respect to the texts used and the extent to which students are asked to interpret and 
think cridcally about what they have read (Kapinus, 2002). 

This paper attempts to answer the inevitable quesdon of whether NAEP and PIRLS are 
measuring the same thing. Using the comparison of NAEP and the 1991 lEA study as a 
model, the NAEP and PIRLS fourth-grade assessments were compared by a group of 
individuals who, coUecdvely, have extensive experience with both assessments, including 
developing the frameworks and items (this group, listed in Appendix A, is hereafter referred 
to as the expert panel). A subset of the expert panel conducted a systemadc comparison of 
the items in both assessments. 

While content is an important consideradon in how the two assessments compare, and is the 
focus of this paper, other aspects of the assessments should be examined as well, including 
how the achievement results are scaled and reported. In this paper, we address the content 
of the assessment, and leave the comparison with respect to other aspects of assessment to 
other researchers to carry out. 

We begin by describing how each assessment defines reading. We examine their formal 
definidons and the aspects of reading they assessed. We then compare the passages in each 
assessment with respect to the kinds of texts as well as their lengths and difficulty. We 
conclude by describing how the assessments compare with respect to types of reading 
processes and skills evaluated in each assessment. These comparisons are based on a 
systemadc classificadon of items by categories in the two frameworks. 

Definitions of Reading 

NAEP’s definidon of reading Literacy is reflected in the following excerpts from the 
framework (NAGB, 2001).^ 

The term reading literacy is not intended to imply only basic or funcdonal literacy. 
Rather the term connotes a broader sense of reading, including knowing when to 
read, how to read, and how to reflect on what has been read. Contemporary research 
indicates that reading is a complex process that involves an interacdon among the 
reader, text, and the context in which something is read. (p. 8) 

Reading for meaning involves a dynamic, complex interacdon among three elements: 
the reader, the text, and the context. The context of a reading situadon includes the 
purposes for reading that the reader might use in building a meaning of the 
text... Good readers bring to this interacdon their prior knowledge about the topic of 
the text and their purposes for reading it, as well as their skill in reading, which 
includes their knowledge about the reading process and the structure of texts, (p. 12) 




‘ Although the NAEP reading framework was “revisited” and updated for the 2003 assessment, resulting in changes to 
the names of categories, because this analysis focuses on the 2002 assessment, the framework categories used for the 
2002 and previously are used. 
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Readers respond to a given text in a variety of ways as they use background 
knowledge and informadon from the text to construct an inidal understanding, 
develop an interpretadon to extend the text's meaning, and examine the meaning so 
as to respond personally and cridcally to the text. (p. 15) 

The PIRLS framework (Campbell et al., 2001) defines reading literacy for PIRLS as 

the ability to understand and use those written forms required by society and/or 
valued by the individual. Young readers can construct meaning from a variety of 
texts. They read to learn, to pardcipate in communides of readers, and for 
enjoyment, (p. 3) 

The PIRLS framework condnues: 

Readers are regarded as acdvely construcdng meaning and as knowing effecdve 
reading strategies and how to reflect on reading. They have posidve atdtudes toward 
reading and read both for recreadon and to acquire informadon. Meaning is 
constructed in the interacdon between reader and text in the context of a pardcular 
reading experience. The reader brings a repertoire of skills, cognidve and 
metacognidve strategies, and background knowledge. The text contains certain 
language and structural elements and focuses on a pardcular topic. The context of 
the reading situadon promotes engagement and modvadon to read, and often places 
specific demands on the reader (p. 3). 

The expert panel noted that there is considerable overlap in how NAEP and PIRLS define 
reading. Both frameworks acknowledge that reading is a construcdve and interacdve process 
involving interacdon between the reader and the text. Both address that the context for 
reading is an important element in how readers make meaning and in the skills and strategies 
they use. Both acknowledge that the structural elements of a text will influence a reader's 
strategies. Authors of the updated NAEP reading framework (2002) also noted that the two 
assessments had similarides. They wrote that the definidons of reading literacy “convey the 
nodon that reading involves developing an understanding of text, thinking about it, and 
using various texts for many different purposes. . .The congruence in framework definidons 
clearly represents a growing internadonal agreement on the important dimensions of reading 
Hteracy" (NAGB 2002, p. 8). 

The panel comparing NAEP and PIRLS also noted differences in what is emphasized in the 
NAEP and PIRLS definidons of reading. The PIRLS definidon is more explicidy targeted to 
younger readers and the reading tasks and processes in which children engage, nodng in the 
definidon the purposes for which young readers read (to learn, to pardcipate in communides 
of readers, and for enjoyment). Of course, PIRLS assessed one populadon, fourth-graders, 
while NAEP assesses three populadons across a broad age and grade span. Another 
difference is the emphasis on readers' response to text: in its definidon, NAEP appears to 
place more emphasis on students' personal response to a text than does PIRLS. NAEP 
devotes an endre category of items to reader-text connecdon (a target of 15% of student 
assessment dme on items classified this way), while PIRLS does not have a separate category 
for this typt of item. 



The NAEP and PIRLS reading assessments are each based on a two-dimensional matrix, 
with the purposes of reading on one dimension and the processes of reading on the other. In both 
assessments, each process is assessed within each purpose for reading. The purposes for 
reading and processes of comprehension assessed by each assessment are shown in Figures 1 
and 2. 

Figure 1: NAEP Framework Dimensions^ 

Purposes for Reading^ 

Processes Reading for Literary Reading for 

Experience Information 

Forming an Initial 
Understanding 



Developing an Interpretation 



Personal Reflection and 
Response 



Demonstrating a Critical 
Stance 



Source: National Assessment of Educational Progress, 2002. 

Figure 2: PIRLS Framework Dimensions 



Processes 



Focus on and Retrieve 
Explicitly Stated Information 



Make Straightforward 
Inferences 



Interpret and Integrate Ideas 
and Information 



Examine and Evaluate 
Content, Language, and 

Textual Elements 

Source: lEA’s Progress in International Reading Literacy Study, 2001 



Purposes for Reading 

Reading for Literary Reading to Acquire 

Experience and Use Information 



^ Although NAEP refers to these as “reading stances” or aspects of “constructing, extending, and examining meaning,” 
they are referred to as “processes” in this document for ease of comparison. 

^ The updated framework (NAGB 2002) uses the following labels for the four process categories: (1) forming a general 
understanding; (2) developing interpretation; (3) making reader-text connections; and (4) examining content and 
structure. 

^ At fourth-grade, NAEP assesses two purposes for reading; at eighth- and twelfth grades NAEP assesses a third 
purpose — Reading to Perform a Task. 
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Purposes for Reading 



Both NAEP and PIRLS assess and report on two purposes for which young readers read,^ 
reading for a literary experience and reading for information. Both call the former purpose as 
‘'reading for literacy experience.’’ NAEP calls its informational category “reading for 
information,” while PIRLS calls it “reading to acquire and use information.” Each 
assessment devotes about half of the assessment to each reading purpose. NAEP devotes 
55% of the assessment to the literary purpose and 45% to the informational purpose and 
PIRLS devotes 50% to each purpose. 

The two Literary purposes are defined in very similar ways. The NAEP framework says: 

Reading for literary experience usually involves the reading of novels, short stories, 
poems, plays, and essays. In these reading situations, readers explore the human 
condition and consider interplays among events, emotions and possibilities. In 
reading for literary experience, readers are guided by what and how an author might 
write in a specific genre and by their expectations of how the text will be organized. 
The readers’ orientation when reading for literary experience usually involves looking 
for how the author explores or uncovers experiences and engaging in vicarious 
experiences through the text. (NAGB, 2001, p. 13) 

The PIRLS framework describes reading for literary experience as follows. 

In literary reading, the reader engages with the text to become involved in imagined 
events, settings, actions, consequences, characters, atmosphere, feelings and ideas, 
and to enjoy language itself. To understand and appreciate literature, the reader must 
bring to the text his or her own experiences, feelings, appreciation of language and 
knowledge of literary forms. For young readers, literature offers the opportunity to 
explore situations and feelings they have not yet encountered, and to experience 
imaginatively an autonomy not yet available to them — Events, actions, and 
consequences depicted in narrative fiction allow the reader to experience vicariously 
and reflect upon situations that, although they may be fantasy, illuminate those of 
real life. The text may present the perspective of the narrator or a principle character, 
or there may be several such viewpoints in a more complex text. Information and 
ideas may be described direcdy or through dialogue and events. Short stories or 
novels sometimes narrate events chronologically, or sometimes make more complex 
use of time with flashbacks or time shifts. (Campbell et al. 2001, p. 17) 

The two assessments both address reading as a means of acquiring information. NAEP 
emphasizes the reading of information texts in order to read and understand informational 
texts for a variety of purposes: 

Reading to be informed usually involves the reading of articles in magazines and 
newspapers, chapters in textbooks, entries in encyclopedias and catalogues, and 
books on particular topics. The type of prose found in such texts has its own 
features and readers need to be aware of those features to understand it... Readers 
read to be informed for different purposes; for example, to find specific pieces of 

^ NAEP also calls the “purposes” “reading situations.” 
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informauon when preparing a research project or to get some general information 
when glancing through a magazine article. (NAGB, 2001, p. 13) 

In addition to addressing reading to acquire informauon, PIRLS also addresses reading to use 
informauon and in that way goes beyond NAEP’s informauonal category (NAEP assesses 
“reading to perform a task’' at eighth- and twelfth-grades). The PIRLS framework says the 
following: 

In reading for information, the reader engages not with imagined worlds, but with 
aspects of the real universe. Through informauon texts, one can understand how the 
world is and has been, and why things work as they do. Readers can go beyond the 
acquisiuon of information and use it in reasoning and in acuon. Informauon texts 
need not be read from beginning to end; readers may select the parts they need. 
(Campbell et al. 2001, p. 17) 

PIRLS does not include a separate reading purpose to address the use of reading to perform 
a task, such as the use of documents like maps or charts or the use of instrucuons on how to 
assemble something because generally young children are not involved in that type of 
reading very often. However, PIRLS does acknowledge that young children do engage in this 
type of reading sometimes by including it in the informational category. The PIRLS 
framework points out that information texts can be organized chronologically or non- 
chronologically (or may include both), which may require the reader to draw upon different 
skills. 

Reading Passages 

The types of reading passages included in NAEP and PIRLS reflect the purposes for reading 
that are assessed. In NAEP, students are presented with short stories, legends, biographies, 
and folktales in the assessment of reading for literary experience. In PIRLS, similarly, 
narrative ficuon in the form of short stories serves as the basis for the PIRLS assessment of 
reading for literary experience. NAEP’s assessment of reading to be informed includes 
magazine articles that focus on people, places, and events of interest to children. Similarly, in 
PIRLS, students are presented with articles about people, places, and events, as well as an 
informational brochure. 

Altogether, the NAEP and PIRLS fourth-grade assessments each comprise eight reading 
passages, four for each purpose. Each student, however, reads and answers questions about 
two passages. Appendix B contains a passage from each assessment; one informauonal text 
from NAEP (administered in 1998) and one literary passage for PIRLS to illustrate the 
nature of the texts used in the two assessments. These passages will be referenced in 
discussions about the assessments to follow. 

Both NAEP and PIRLS strive to have the assessment be an “authentic” reading experience 
for students. That is, both try to present tasks that are similar to what students would 
actually encounter when reading in and out of school. The NAEP framework calls for the 
use of “authentic” texts. AU passages were previously published in children’s magazines or 
other pubHcauons and generally are not edited at all for use in NAEP. Also, the format is 
kept as close to the original as possible. In selecting passages, NAEP uses passages that are 
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like those that students encounter in and out of school, but that were published long enough 
ago that the students taking the assessment probably have not read the particular passages. 

PIRLS also strives to use authentic, previously published texts. However, PIRLS has a more 
liberal policy on editing and changing the format of the texts that are used. Largely because 
of the constraints inherent in an assessment that is administered across languages and 
cultures, PIRLS did change some of the original language in the passages so that it would be 
acceptable to a more diverse audience, could be more easily translated and retain the same 
meaning, and could meet requirements for length. 

One feature of the PIRLS assessment that speaks to authenticity is the use of the 'TIRLS 
Reader.” While most students in the assessment get a test booklet that contains two 
sections, each with one reading passage and 11-14 questions about the passage, some 
students receive a color booklet with two passages (one literary and one informational) and 
the accompanying items in a separate test booklet. The purpose of this layout was to evoke a 
more authentic reading experience for students. About a quarter of the students in the 
PIRLS assessment were administered the PIRLS Reader. 

How Interchangeable are the NAEP and PIRLS Reading Passages? 

After reading all of the passages in both assessments, the expert panel concluded that most 
of the NAEP passages could appear on PIRLS and most of the PIRLS passages could 
appear on NAEP, although overall the PIRLS passages were shorter and appeared less 
complex than the NAEP passages. This judgment was made without the benefit of word 
counts or results of the empirical analyses. 

For both NAEP and PIRLS, there were two passages that the panel felt would not appear 
on the other assessment. The panel felt that two of the eight NAEP passages would be 
inappropriate for PIRLS because of their length and complexity. Both of these texts (one a 
short story and one an informational article) were included in NAEP at both fourth- and 
eighth grades and were thus the more difficult of the NAEP fourth-grade passages. 

Of the eight PIRLS passages, the panel felt that all but two would be appropriate for NAEP. 
One of the two that the panel felt would not appear on NAEP was an informational 
brochure that the group felt was more appropriate for assessing “reading to perform a task” 
than “reading to be informed,” and therefore might appear on NAEP at eighth grade where 
“reading to perform a task” is assessed. The other passage was considered inappropriate 
because it had been excerpted from a book and revised substantially and therefore lacked the 
authenticity required by NAEP. 

Length 

The panel noted in their review of the NAEP and PIRLS passages that the PIRLS passages 
were, on average, shorter. Indeed, the NAEP reading passages are, on average, twice as long 
as the PIRLS reading passages. As shown in Table 1, NAEP passages range in length from 
691 to 1365 words, with an average word length of 1000. In contrast, PIRLS passages range 
in length from 293 (a brochure) and 322 (mainly continuous text) to 804 words, with an 
average of 547 words. The passages included in Appendix B illustrate the difference in 
length. The PIRLS story, Upside-Down Mice, has 521 words, close to the average for the 
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PIRLS passages. The NAEP passage, 'Blue Crabs (administered in 1998 and not included in 
the counts in Table 1), is an informadonal ardcle that is 880 words, a typical length for the 
NAEP fourth-grade passages. 

Table 1: Number of Words in NAEP and PIRLS Fourth-Grade Reading Passages 





Number of Words in Each Passage 


Average 


NAEP Passages 


691, 790, 851, 946, 1038, 1120, 1196, 1365 


1000 


PIRLS Passages 


293, 322, 521, 537, 546, 621, 730, 804 


547 



Source: National Assessment of Educational Progress, 2002 Assessment; lEA’s 
Progress in International Reading Literacy Assessment, 2001 Assessment. 



Passage Difficulty 

While the panel of experts recognized differences in difficulty between the NAEP and 
PIRLS passages, and noted the differences in length, they wanted to provide a more 
empirically-based measure of difficulty. Two widely used readability formulas, the Fry and 
the Flesch, were used to obtain measures for the NAEP and PIRLS fourth-grade reading 
passages. In addition, a Lexile analysis was conducted to obtain an additional measure that 
indicates the reading demand of the text in terms of the semantic difficulty (vocabulary) and 
syntactic complexity (sentence length). 

Readability 

For the Fry and Flesh readability formulas, the number of words and syllables in each of 
three 100-words samples (per text) are counted and used as the basis for determining an age 
and grade level for the text. See Appendix C for a description of the formulas and 
procedures used. Counting the number of sentences or syllables per 100 words is quite 
straightforward and easily replicable across raters and across time. However, readability 
formulae go one step further. They associate length of words and sentences with increasing 
difficulty that can then again be associated with the increasing skill that is accrued with age 
or grade level. This connection is derived from materials used in classrooms at a particular 
point in time. The strength of this association was used as a predictor of the probability that 
other materials with the same degree of characteristics would then be appropriate at the age 
or grade level as well. 

Counts of the number of syllables and sentences per 100 words provide some evidence that 
the NAEP passages are, overall, more complex. As shown in Table 2, NAEP has, on 
average, more syllables per 100 words than PIRLS (138.2 compared with 133.6), suggesting 
the use of longer and perhaps more unfamiliar words in the NAEP passages. NAEP has an 
average of 6.8 sentences per 100 words, compared with 8.6 sentences for PIRLS, indicating 
that the NAEP sentences are longer (also in Table 2). The difference in the average number 
of sentences suggests that the NAEP texts are more complex, with more embedded clauses 
and a more complex syntactical structure. 
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Table 2: Average Number of Sentences and Syllables per 100 Words, NAEP and PIRLS 
Reading Passages 





Sentences 


Syllables 


NAEP 


6.8 


138.2 


PIRLS 


8.6 


133.6 



Note: See Appendix C for details on analyses 



Based on the Fry analysis, as shown in Table 3, predictions could be made that on average 
the NAEP passages would be appropriate for grade 7 (and average age of 12.3) and PIRLS 
for grade 5 (and average age of 10.4). Similarly, the Flesch analysis indicated that the NAEP 
passages would be appropriate for grade 7 and PIRLS for grade 5-6. The Flesch analysis 
provides a ‘‘reading ease’’ score rather than an age, and it too shows that NAEP is more 
difficult; N AE P was rated “fairly easy” while PIRLS was rated “easy.” Appendix C presents 
results of the Fry and Flesch analyses on each of the NAEP and PIRLS reading passages. 

Overall, both formulae indicate about a two grade-level spread between NAEP, which is 
more difficult, and PIRLS. Therefore, the panelists’ impression was confirmed by a less 
subjective measure. 

Table 3: Average Age and Grade Level Determined by Fry and Flesch Readability 
Analyses: PIRLS and NAEP Fourth Grade Reading Passages 



Fry Analysis Flesch Analysis 





Average 

Age 


Average 

Grade 

Level 


Average 

Reading 

Ease 


Average 

Grade 

Level 


NAEP 


12.3 


6.9 


Fairly Easy 
(74.5) 


7th 


PIRLS 


10.4 


5.0 


Easy (81.7) 


5th to 
6th 



Note: See Appendix for details on analyses 



Although the results of the readability analyses support the panel’s observation that the 
NAEP passages were more difficult than those in PIRLS, there are some caveats associated 
with readability formulae. Historically, readability formulae have been used to provide a 
quick estimate of the difficulty and grade appropriateness of a text. In general, as a class of 
tools they were good predictors for these purposes. However, as more and more states 
moved towards stringent requirements related to the evaluation of grade appropriateness for 
textbook adoption, these indicators were found to be insufficient for the expanded purpose 
and fell into disfavor. The reading formulae did not account for a sufficient number of 
salient text features that figure prominendy in comprehension, such as variation in 
vocabulary, topic, and appeal. 
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Lexile Analysis 



Whether the NAEP and PIRLS passages are or are not appropriate for fourth-graders could 
not be concluded based on older readability formulae not only because of their more limited 
measurement, but also because the norming information that is the basis of the two 
readability formulae was established over twenty-five years ago. To address this issue a more 
current measure of passage difficulty was obtained using the Lexile system (see Appendix C 
for description of procedure). The Lexile provides a measure of difficulty that takes into 
account semantic difficulty (vocabulary) and syntactic complexity (sentence length) and 
assigns an appropriate grade level. Developed and normed in recent years, Lexiles provide a 
more accurate measure of grade appropriateness. As shown in Table 4, the NAEP passages 
were determined to be appropriate for 4* to 5^ grade and the PIRLS passages appropriate 
for 3""^ to 4^*" grade. This is consistent with the judgment of the expert panel. 

Table 4: Lexile Score and Corresponding Grade Levels for PIRLS and NAEP Fourth Grade 
Reading Passages 





Lexile Score 


Corresponding Grade 
Level 


NAEP Passages 


818.8 


4* or 5th 


PIRLS Passages 


695.7 


3"' or 4th 



Note: See Appendix C for details on analyses 



Distribution of Item Types in NAEP and PIRLS 

In both NAEP and PIRLS, students respond to multiple-choice and constructed-response 
questions (short answer and extended response) based on the passages, and the distribution 
of these item types are similar in both assessment (Table 4). In each assessment, close to half 
of the items are in multiple-choice format, with four response options (45% of NAEP items 
and 47% of PIRLS items). Both NAEP and PIRLS include constructed-response items that 
are scored for two, three, or four levels of correctness (including incorrect), with ''short 
constructed-response” items scored for two or three levels and "extended constructed 
response” items scored for four levels. Each item has a scoring guide that delineates the 
requirements for different levels. Scoring guides are used by scorers to make reliable scoring 
decisions. The percentages of short and extended constructed-response items in each 
assessment are comparable: In NAEP, 45% of the items are short-constructed response and 
10% are extended; in PIRLS, 44% of the items are short-constructed response and 8% are 
extended. 
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Table 4: Distribution of Item Types in NAEP and PIRLS, based on Percentage of the 
Number of Items in Each Assessment 





Multiple-choice 


Short Constructed 
Response 


Extended 

Constructed 

Response 


NAEP (Fourth- 


45% 


45% 


10% 


grade) 








PIRLS 


47% 


44% 


8% 



Source: National Assessment of Educational Progress, 2002 Assessment; lEA's 
Progress in International Reading Literacy Assessment, 2001 Assessment. 



Reading Processes Assessed by NAEP and PIRLS 

While the purposes for reading and the types of texts used form the content of the 
assessment, the reading processes define the skills and abilities that students must draw on in 
response to the texts. Each item is written to address one process. Both NAEP and PIRLS 
have four categories of reading processes; each is described briefly below. 

NAEP Processes Categories 

Forming an Initial Understanding: “requires the reader to provide an initial 
impression or global understanding of what was read. It involves considering the text 
as a whole or in a broad perspective. . 

Developing an Interpretation: “...requires the reader to go beyond the initial 
impression to develop a more complex understanding of what was read. It involves 
linking information across parts of a text as well as focusing on specific 
information.. 

Personal Reflection and Response: “... require [s] the reader to connect knowledge 
from the text with his or her personal background knowledge. The focus here is on 
how the text relates to personal knowledge.” 

Developing a Critical Stance: “...requires the reader to stand apart from the text and 
consider it objectively. It involves a range of tasks including such behaviors as critical 
evaluation, comparing and contrasting, application to practical tasks, and 
understanding the impact of such text features as irony, humor, and organi 2 ation.” 
(Reading Framework 2001, pp 15-17) 

PIRLS Process Categories 

Focus on and Retrieve Exphddy Stated Information: “Successful retrieval requires a 
fairly immediate or automatic understanding of the text. This process requires htde 
or no inferring or interpredng. There are no ^^gaps” in meaning to be filled — the 
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meaning is evident and stated in the text. . ..Focus on the text typically remains at the 
sentence or phrase level in this type of processing.’’ 

Make Straightforward Inferences: ‘‘Making inferences allows the reader to move 
beyond the surface of texts and to fill in the ‘‘gaps” in meaning that often occur in 
texts. ... Although the ideas may be explicidy stated, the connecdon between them is 
not, and thus must be inferred. Straightforward inferences are very much text- 
based.” 

Interpret and Integrate Ideas and Informadon: “. . .the reader is processing text 
beyond the phrase or sentence level. . . .When they interpret and integrate text 
informadon and ideas, readers may need to draw on their background knowledge 
and experience more than they do for straightforward inferences. . . .By engaging in 
this interpredve process, readers are attempdng to construct a more specific or more 
complete understanding of the text by integradng personal knowledge and 
experience with meaning that resides in the text.” 

Examine and Evaluate Content, Language, and Textual Elements: “The reader 
engaged in this process is standing apart from the text and examining or evaluadng it. 
The text content, or meaning, may be examined from a very personal perspecdve or 
with a cridcal or objecdve view. . .In examining or evaluadng elements of text 
structure and language, readers draw upon their knowledge of language usage and 
general or genre-specific features of texts.” (Campbell et al. 2001, pp 10—15) 

The expert panel examined the descripdons of the categories and discussed the apparent 
similarides and differences among them in an attempt to determine where categories might 
overlap in what they assessed. Their inidal impressions of how the categories compared, 
prior to systemadcally classifying the items, suggested that overall the assessments were 
calling on similar reading skills, but that they do not break out and describe the skills in the 
same manner in the framework. The panel noted similarides between NAEP’s “Developing 
an interpretadon” and PIRLS’s “Make straightforward inferences” and “interpret and 
integrate informadon and ideas” and some overlap between NAEP’s “demonstradng a 
cridcal stance” and PIRLS’s “Examine and evaluate content, language, and textual 
elements.” 

The panel’s inidal impression that the two assessments are, overall, assessing the same 
domain was not upheld endrely when it delved more deeply into the content by examining 
and classifying the items. Through this exercise, described below, the panel determined that 
while there are many similarides in what is measured on each assessment, there also are 
disdnct differences. 

First, in Figures 1 and 2 we present items from NAEP and PIRLS, respecdvely, to illustrate 
how the different reading processes are assessed (passages are included in Appendix B). The 
example NAEP passage was administered in 2000. 
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Figure 1: Example NAEP Fourth-grade Items 



Students read an informational passage, A Brick to Cuddle Up To, about how people kept warm during 
colonial times. See Appendix. B. 

Developing an initial understanding 

1) You would probably read this article if you wanted to know how the colonists 

A) cooked their food 

B) traveled in the winter 

C) washed their clothes 

*D) kept warm in cold weather. 



Developing an interpretation 

2) A colonist would probably have used a foot stove when 
*A) going on a trip 

B) sleeping in a bed 

C) sitting by the fireplace 

D) working around the house 

Source: National Assessment of Educational Progress, 2000 Assessment. 




22 



16 



Students read an informational passage, Blue Crabs^ describing the experiences of huntingfor and 
catching blue crabs. It includes infou/iation about the crabs' appearance, habits, habitats, and survival 
techniques, and tvhat it is like to hunt for them. See Appendix B 



Developing an interpretation 

3) Why does a blue crab hide after molting? 
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Personal reflection and response 



4) What is the most interesting thing you learned from the passage about blue crabs? 
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Demonstrating a critical stance 

The author of the article helps you to learn about blue crabs by 

A) explaining why they are an endangered species 

B) comparing them to other arthropods 

C) discussing their place in the food chain 

*D) providing details about their unique characteristics 



Source: National Assessment of Educational Progress, 1998 Assessment. 
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Figure 2: Example PIRLS Fourth-grade Items 



Students read a short story by Koald Dahl called ‘Dpside-down Mice, ** In the story ^ an elderly man with 
mice in his house carries out a plan to trick the mice and get rid of them. The mice observe the man as he 
carries out his plan, unaware of their fate. See Appendix B. 

Focus on and retrieve explicitly stated information 

1) Where did Labon put the mousetraps? 

A) in a basket 

B) near the mouseholes 

C) under the chairs 
*D) on the ceiling 

Make straightforward inferences 

2) Why did Labon want to get rid of the mice? 

A) He had always hated mice, 

*B) There were too many of them. 

C) They laughed too loudly 

D) They ate all his cheese. 

Interpret and integrate ideas and information 

3) You learn what Labon is like from the things he does. Describe what he is like, and 
give two examples 

Labon is tricky because he glued the mousetraps to the ceiling on the first ni^ht just to make 
them think he was silly. Then he ^lued the furniture to the ceiling to make them think they 
were upside down 

Examine and evaluate content, language, and textual elements 

4) Think about what Labon and the mice did in the story. Explain what makes the 
story unbelievable. 

No one would be able to glue furniture to the ceiling like Labon did 

Source; lEA’s Progress in International Reading Literacy Assessment, 2001 
Assessment. 
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To What Extent Do NAEP and PIRLS Measure Similar Skills? 



To understand the extent to which each assessment measured aspects of reading 
comprehension measured by the other, the 82 NAEP and 98 PIRLS items were classified by 
the panel by the process categories in each other’s frameworks. That is, each NAEP item 
was classified by one of the four PIRLS process categories and each PIRLS item was 
classified by one of the four NAEP process categories. The results of this cross-classification 
reveal considerable overlap in what the two assessments are assessing, yet also some striking 
differences in what each emphasizes. 

Table 5 shows the distribudon of items in each assessment by the PIRLS classificadon 
scheme. The first row, NAEP items, shows the percentage of NAEP items in each PIRLS 
category, according to the expert panel. The second row, PIRLS items, shows the percentage 
of items in each category according to the PIRLS assessment developers.^ 

Cross-Classification of Items by PIRLS Framework 

Both assessments have similar levels of emphasis on students’ ability to make 
straightforward inferences. In NAEP and PIRLS, a litde more than a quarter of the items 
(27% for NAEP and 28% for PIRLS) assess this skill. The two assessments do not place 
similar levels of emphasis on the other PIRLS process categories. For example, NAEP 
places more weight on complex interpretadon than does PIRLS. Nearly half of NAEP items 
(46%) were classified as “Interpret and integrate ideas and informadon,” while about one- 
third (32%) of PIRLS items were. Compared to NAEP, PIRLS places greater emphasis on 
examining and evaluadng content, language, and textual elements (15% in PIRLS compared 
with 9% in NAEP). PIRLS also places more emphasis on focusing on and retrieving 
informadon than NAEP — 26% of the PIRLS items were classified “Focus on and retrieve 
explicidy stated informadon” compared with 18% of NAEP items. 

Table 5: NAEP and PIRLS Items Classified by PIRLS Framework 





Focus on and 
retrieve explicitly 
stated information 


Make 

straightforward 

inferences 


Interpret and 
integrate ideas and 
information 


Examine and evaluate 
content, language, and 
textual elements 


NAEP 

Items* 


18% 


27% 


46% 


9% 


PIRLS 

Items** 


26% 


28% 


32% 


15% 



*As classified by expert panel. 

**As classified by PIRLS developers. 

The cross-classificadon exercise involved having the panel members classify the PIRLS items 
by the PIRLS categories, although the PIRLS classificadons reported in Table 5 are the 
PIRLS item classificadons according to the developers of the PIRLS assessment (the 
“actual” item classificadons). But, to what extent did the panel agree with the classificadons 
of the PIRLS items? Table D.l in the appendix shows panel had a nearly idendcal 



^ Classification of PIRLS items was provided by the PIRLS International Study Center, Boston College. 



distribudon of items across the PIRLS categories when it classified the PIRLS items by the 
PIRLS framework. This suggests that each category in PIRLS is disdnct from every other 
category, with litde overlap and also that the PIRLS framework was explicit enough to 
communicate the types of skills and processes subsumed under each category. 

Cross-Classification of Items by NAEP Framework 

Table 6 shows the distribudon of items in each assessment according to the NAEP 
classificadon scheme. Again, we see differences in what each assessment emphasizes. Few 
PIRLS items fit into “Forming an inidal understanding” (4%) and “Personal reflecdon and 
response” (3%), while 10% and 15% of NAEP items were so classified by NAEP 
developers. While 12% of PIRLS items were classified as “Demonstradng a cridcal stance,” 
21% of NAEP items were classified this way. The area in which more PIRLS items than 
NAEP items were classified was “Developing an interpretadon” (62% compared with 55%). 

The most striking outcome of this exercise was that it was not possible to fit 18% of the 
PIRLS items into the NAEP framework. These were items that asked students to locate 
informadon in the text that was virtually an idendcal match between the item and the text. 
These items are shown in the table in the category labeled “other.” It is expected that such 
items would be easier for students, although other factors beyond the match between the 
item stem and the text may influence difficulty, and these features were not examined. We 
can conclude, however, that 18% of PIRLS items are unlike those that appear on NAEP 
since NAEP does not include items of this nature. 

An example of this type of item is shown in Figure 2 (page 17) as example item 1. There, 
students are asked to locate a specifically stated piece of informadon and although they need 
to idendfy the appropriate area of the text and determine which piece of informadon is 
relevant, the informadon is explicidy stated and there is ho need for interpretadon. 



Table 6: NAEP and PIRLS Items Classified by NAEP Framework 





Forming an Initial 
understanding 


Developing an 
interpretation 


Personal reflection 
and response 


Demonstrating a 
critical stance 


Other 


NAEP 

Items* 


10% 


55% 


15% 


21% 


0% 


PIRLS 

Items** 


4% 


62% 


3% 


12% 


18% 



*As classified by NAEP assessment developers. 

**As classified by panel. 

The cross-classificadon exercise involved having the panel members classify the NAEP 
items by the NAEP categories, although the classificadons reported in Table 5 are the 
NAEP item classificadons according to NAEP developers (the “actual” item classificadons). 
But, to what extent did the panel agree with the classificadons of the NAEP items? Unlike 
the close agreement between the panel and PIRLS assessment developers (Table D.l in 
Appendix D), the panel was not in full agreement with the “actual” NAEP item 
classificadons. As shown in Table D.l, the panel classified more items “Developing an 
interpretadon” and fewer items “Demonstradng a cridcal stance” than the NAEP 
developers. This suggests that there is some overlap in what these categories are assessing 
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and perhaps some ambiguity in how they are described and communicated in the 
framework. It should be noted that in coming to consensus on how each item should be 
classified, the panel found it more difficult to agree with each other on the NAEP items, 
than on the PIRLS item classifications, and that could be contributing to the disagreement 
with the ‘‘actuaf’ classifications. 

Detailed View of Cross-Classification 

In order to more fuUy understand how the NAEP and PIRLS assessments compare, the 
following tables show more detailed information about how the expert panel classified the 
items in each. Taken together, the distributions suggest that the NAEP process categories 
are broader than those in PIRLS. Items from a category in the NAEP framework tend to be 
dispersed across the PIRLS categories, while items from a category in the PIRLS framework 
tend to fall fairly cleanly into one NAEP category. 

Cross-Classification of Items by PIRLS Framework 

Table 7 shows the items classified as NAEP’s “Forming an initial understanding’’ distributed 
across PIRLS categories according to how the committee classified the items during the 
cross-classification exercise: 13% were classified by the panel as “Focus on and retrieve 
explicitly stated information,” 25% were classified “Make straightforward inferences,” 63% 
were classified “Interpret and integrate ideas and information,” and 0% were classified 
“Examine and evaluate content, language, and textual elements.” This suggests that the 
“Forming an initial understanding” category in NAEP includes items assessing retrieval, 
inference, and interpretation skills. 

Table 7: NAEP “Forming an initial understanding” Items Classified by PIRLS Framework 
Categories 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


Focus on and retrieve explicitly stated 
information 


13% 


Make straightforward Inferences 


25% 


Interpret and integrate ideas and 
information 


63% 


Examine and evaluate content, 
language, and textual elements 


0% 


Total 


100% 
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Similar to items in the ‘Terming an initial understanding category/’ NAEP items classified 
as “Developing an interpretadon” are dispersed across the PIRLS process categories (Table 
8). About one-third of NAEP items fell into each of three PIRLS categories, and 2% in 
“Examine and evaluate content, language, and textual elements,” suggesting that this 
category includes items tapping a range of skills, including retrieval of information. 

Table 8: NAEP “Developing an interpretation” Items Classified by PIRLS Framework 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


Focus on and retrieve explicitly stated 
information 


29% 


Make straightforward inferences 


36% 


Interpret and integrate ideas and 
information 


33% 


Examine and evaluate content, 
language, and textual elements 


2% 


Total 


100% 



NAEP items classified as “Personal reflection and response” are not as dispersed across the 
PIRLS categories to the extent that items in the previous two categories are, suggesting that 
this is a more tightly defined category: three-quarters (75%) of these items were classified as 
“Interpret and integrate ideas and information.” Close to 20% of these items were deemed 
to fit into “Examine and evaluate content, language, and textual elements.” None of the 
items were classified as “Focus on and retrieve explicitly stated information” reflecting the 
fact that these items are not asking students to retrieve information from the text but rather 
to think about what they have read in reference to their own experiences or points of view. 
See Table 9. 



Table 9: NAEP “Personal reflection and response” Items Classified by PIRLS Framework 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


focus on and retrieve explicitly stated 
information 


0% 


make straightforward inferences 


8% 


interpret and integrate ideas and 
information 


75% 


examine and evaluate content, 
language, and textual elements 


17% 


Total 


100% 
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NAEP ‘‘Demonstradng a critical stance’’ items (Table 10) fall across the four PHILS 
categories, suggesting that it too is a broad category of skills, including retrieving, making 
inferences, and examining and evaluating the text, and an emphasis (about half of the items) 
on requiring students to interpret and integrate ideas and information. 

Table 10: NAEP “Demonstrating a critical stance” Items Classified by PIRLS Framework 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


Focus on and retrieve explicitly stated 
information 


6% 


Make straightforward inferences 


18% 


Interpret and integrate ideas and 
information 


53% 


Examine and evaluate content, 
language, and textual elements 


24% 


Total 


100% 



Cross-Classification of Items by NAEP Framework 

In contrast to the way NAEP items from a single NAEP category were distributed across 
the PIRLS categories for three of the four NAEP categories, the PIRLS items tend to fit 
clearly into one or two NAEP categories (or not in any). As shown in Table 11, about two- 
thirds of the PIRLS ‘Tocus on and retrieve expHcidy stated informadon and ideas” items did 
not fit a NAEP category and were therefore classified as ‘‘Other.” These were items that 
required students to make virtually an exact match between the item stem and text. Most of 
the remaining items were classified as “Developing an interpretadon,” suggesdng that there 
is some overlap between this NAEP category and PIRLS’ “Focus on and retrieve.” 
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Table 11: PIRLS ^Tocus on and retrieve explicitly stated information and ideas” Items Classified by 
NAEP Framework 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


Forming an initial understanding 


4% 


Developing an interpretation 


28% 


Personal reflection and response 


0% 


Demonstrating a critical stance 


0% 


Other (not a NAEP category) 


68% 


Total 


100% 



Almost 90% of PIRLS ‘'Make straightforward inferences” items were classified as 
“Developing an interpretation,” indicating an overlap between the skills assessed by these 
two categories. Some of these items (11%) tap skills assessed by the “Demonstrating a 
critical stance” category. 

Table 12: PIRLS “Make straightforward inferences” Items Classified by NAEP Framework 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


Forming an Initial understanding 


0% 


Developing an interpretation 


89% 


Personal reflection and response 


0% 


Demonstrating a critical stance 


11% 


Total 


100% 
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Nearly all PIRLS ‘‘Interpret and integrate” items were classified as NAEP's “Developing an 
interpretation,” indicating an overlap in the skills assessed by items in these two categories 
(Table 13). 

Table 13: PIRLS “Interpret and integrate ideas and information” Items Classified by NAEP 
Framework 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


Forming an initial understanding 


0% 


Developing an interpretation 


94% 


Personal reflection and response 


3% 


Demonstrating a critical stance 


3% 


Total 


100% 



PIRLS items classified as “Examine and evaluate content, language, and textual elements” 
cut across the NAEP process categories, with some items in each, although 53% were 
classified as “Demonstrating a critical stance” (Table 14). 

Table 14: PIRLS “Examine and evaluate content, language, and textual elements” Items Classified 
by NAEP Framework 



PIRLS Process Category 


Percent of Items 
According to 
Panel 


forming an initial understanding 


20% 


developing an interpretation 


7% 


personal reflection and response 


13% 


demonstrating a critical stance 


53% 


Other (not a NAEP category) 


7% 


Total 


100% 
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Conclusion 



The comparison of the NAEP and PIRLS fourth-grade reading assessments suggests that 
there is a great deal of overlap in what the two assessments are measuring. However, while 
they do seem to define reading in similar ways. PIRLS has more text-based tasks and shorter, 
less complex reading passages than NAEP. The similarities and differences between the two 
are described below. 

This comparison revealed that overall, the NAEP and PIRLS reading assessments are quite 
similar. Both define “reading” similarly, as a constructive process. Both use high-quahty 
reading passages and address similar purposes for which young children read — for literary 
experience and information. Both call for students to develop interpretations, make 
connections across text, and evaluate aspects of what they have read. Finally, both have a 
similar distribution of multiple-choice and constructed-response items: in each, about half of 
the items are constructed-response items. 

While the two assessments have similar definitions of reading and assess many of the same 
aspects of reading, a closer look at how the domain is operationalized by each revealed that 
there are some important differences. NAEP places more emphasis than PIRLS on having 
students take what they have read and connect to other readings or knowledge. This is 
consistent with the value placed on readers’ response to text when the NAEP reading 
framework was developed. 

PIRLS places a greater emphasis than NAEP on text-based reading skills and interactions, 
including items that ask students to locate information in the text, make text-based 
inferences and interpretations, and evaluate aspects of the text. Perhaps the most striking 
difference is that close to 20% (18%) of the items in PIRLS require students to locate 
information in the text that is virtually an identical match to what is in the stem of the item. 
NAEP does not have any items requiring a verbatim match. NAEP does have items asking 
students to locate specific information or ideas in the text, but the information is not a 
verbatim match. 

The PIRLS reading passages are, on average, about half the length of the NAEP reading 
passages. PIRLS passages are, on average, about 547 words, while NAEP passages are, on 
average, about 1000 words. Readability formulas indicate that the passages used in PIRLS are 
less complex than those used in NAEP (one to two grade levels lower, on average). 

The classification of items also revealed differences in how the two frameworks function. 
The panel had an easier time classifying PIRLS and NAEP items by the PIRLS framework 
categories than by the NAEP framework categories. When classifying PIRLS items by the 
PIRLS framework the panel had almost 100% agreement with the lEA assessment 
developers. When classifying by the NAEP framework, however, there was more 
disagreement as to the categories in which NAEP and PIRLS items should be classified. 
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Appendix B: Example Passages 

NAEP Example Passages 







By George W. Frame 

Nearly every day last summer my nephew Keith and I went crabbing in a creek on 
the New Jersey coast. We used a wire trap baited with scraps of fish and meat. 
Each time a crab entered the trap to eat, we pulled the doors closed. We cooked 
and ate the crabs we caught. 

Blue crabs are very strong. Their big claws can make a painful pinch. When 
cornered, the crabs boldly defend themselves. They wave their outstretched claws 
and are fast and ready to fight. Keith and I had to be very careful to avoid having 
our fingers pinched. 

Crabs are arthropods, a very large group of animals that have an external 
skeleton and jointed legs. Other kinds of arthropods are insects, spiders, and 
centipedes. Blue crabs belong to a particular arthropod group called crustaceans. 
Crustaceans are abundant in the ocean, just as insects are on land. 

The blue crab’s hard shell is a strong armor. But the armor must be cast off from 
time to time so the crab can grow bigger. Getting rid of its shell is called molting. 

Each blue crab molts about twenty times during its life. Just before molting, a 
new soft shell forms under the hard outer shell. Then the outer shell splits apart, 
and the crab backs out. This leaves the crab with a soft, wrinkled, outer covering. 
The body increases in size by absorbing water, stretching the soft shell to a much 
larger size. The crab hides for a few hours until its new shell has hardened. 

Keith and I sometimes found these soft-shell crabs clinging to pilings and hiding 
beneath seaweed. 

Blue crabs mate when the female undergoes her last molt and still has a soft shell. 
The male courts her by dancing from side to side while holding his claws 
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outstretched. He then transfers sperm to the female, where they are 
stored until egg laying begins several months later The female blue 
crab mates only once but receives enough sperm to fertilize all the 
eggs that she will lay in her lifetime. Usually she lays eggs two or three 
times during the summer, and then she dies. 

When the eggs are fertilized and laid, they become glued to long 
hairs on the underside of the female's abdomen. The egg mass 
sometimes looks like an orange-brown sponge and contains up to two 
million eggs until they hatch — about nine to fourteen days later. Only 
one of the blue crabs that we caught last summer was carrying eggs, 
and we returned her to the water so her eggs could hatch. Most 
females with eggs stay in the deeper, saltier water at the ocean's edge 
rather than in the marshes. 

The young blue crabs, and most other young crustaceans, hatch into 
larvae that look very different from their parents. The tiny blue crab 
babies are hardly bigger than a speck of dust. They are transparent 
and look like they are all head and tail. These larvae swim near the 
surface of the sea, and grow a new and bigger shell every few days. 
They soon change in shape so that they can either swim or crawl 
around on the bottom. Then they molt again and look like tiny adult 
crabs. After that their appearance does not change, but they continue 
to molt every twenty or thirty days as they grow. 

i 

As blue crabs become older, some move into shallower waters. The males in 
particular go into creeks and marshes, sometimes all the way to the freshwater 
streams and rivers. Keith and I caught ninety-two blue crabs in the shallow creek of 
the tide marsh last summer. Eighty-seven of those crabs were males, and only five 
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Gulls find and eat many blue crabs. They easily catch crabs that hide in puddles 
at low tide. Other predators are raccoons, alligators, and people. If caught, the 
crabs sometimes drop off a leg or claw to escape. Seven of the blue crabs that 
Keith and I caught were missing a claw. 

Crabs are able to replace their lost limbs. If a leg or claw is seriously injured, the 
crab drops it off. The opening that is left near the body closes to prevent the loss of 
blood. Soon a new limb begins growing at the break. The next time the crab molts, 
the tiny limb’s covering is cast off, too, and the crab then has a new usable leg or 
claw. The new limb is smaller that the lost one. But by the time the crab molts two 
or three more times, the new leg or claw will be normal size. 

Many fishermen catch crabs to sell. Most are caught in wire traps or with baited 
lines during the summer while the crabs are active. In the winter, the fishermen 
drag big nets through the mud for the dormant crabs. Commercial fishermen catch 
a lot of crabs, sometimes more than 50 million pounds in a year. And many other 
crabs are caught by weekend fishermen who crab for fun and food. 

The blue crab has a scientific name, just like all other living things. Its name is 
Callinectes sapidus. In the Latin language Callinectes means "beautiful swimmer," 
and sapidus means "delicious." I think that scientists gave the blue crab a very 
appropriate name. 

W0000016 

Used by permission of Highiights for Chiidren, Inc., Columbus, OH. © 1988. 




Imagine shivering on a cqld winter’s night. The tip of your nose tingles in the frosty air. 
Finally, you climb into bed and find the toasty treat you have been waiting for-your very 
own hot brick. 

I If you had lived in colonial days, that would not sound as strange as it does today. 
Winters were hard in this New World, and the colonists had to think of clever ways to 
fight the cold. At bedtime, they heated soapstones, or bricks, in the fireplace. They 
wrapped the bricks in cloths and tucked them into their beds. The brick kept them warm 
at night, at least for as long as its heat lasted. 
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at night, at least for as long as its heat lasted. 

Before the colonists slipped into bed, they rubbed their icy sheets with a bed warmer. 
This was a metal pan with a long wooden handle. The pan held hot embers from the 
fireplace. It warmed the bedding so well that sleepy bodies had to wait until the sheets 
cooled before climbing in. 

Staying warm wasn’t just a bedtime problem. On winter rides, colonial travelers covered 
themselves with animal skins and warm blankets. Tucked under the blankets, near their 
feet, were small tin boxes called foot stoves. A foot stove held burning coals. Hot smoke 
puffed from small holes in the stove’s lid, soothing freezing feet and legs. When the 
colonists went to Sunday services, their foot stoves, furs, and blankets went with them. 
The meeting houses had no heat of their own until the 1800s. 

At home, colonial families huddled close to the fireplace, or hearth. The fireplace was 
wide and high enough to hold a large fire, but its chimney was large, too. That caused a 
problem: Gusts of cold air blew into the house. The area near the fire was warm, but in 
the rest of the room it might still be cold enough to see your breath. 

Reading or needlework was done by candlelight, or by the light of the fire. During the 
winter, animal skins sealed the drafty windows of some cabins and blocked out the 
daylight. The living area inside was gloomy, except in the circle of light at the hearth. 

Early Americans did not bathe as often as we do. When they did, their "bathroom” was 
the kitchen, in that toasty space by the hearth. They partially filled a tub with cold water, 
then warmed it up with water heated in the fireplace. A blanket draped from chairs for 
privacy also let the fire's warmth surround the bather. 

The household cooks spent hours at the hearth. They stirred the kettle of corn pudding 
or checked the baking bread while the rest of the family carried on their own fireside 
activities. So you can see why the fireplace was the center of a colonial home. 

The only time the fire was allowed to die down was at bedtime. Ashes would be piled 
over the fire, reducing it to embers that might glow until morning. 

By sunrise, the hot brick had become a cold stone once more. An early riser might get 
dressed under the covers, then hurry to the hearth to warm up. 

Maybe you'd enjoy hearing someone who kept warm in these ways tell you what it was 
like. You wouldn’t need to look for someone who has been living for two hundred years. 
In many parts of the country the modern ways didn’t take over from the old ones until 
recently. Your own grandparents or other older people might remember the warmth of a 
hearthside and the joy of having a brick to cuddle up to. 

Used by permission of Highlights for Children, Inc., Columbus, OH 
Copyright ©1991. Illustration by Katherine Dodge. 
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PIRLS Example Passage 
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Upside-Down Mice | 

by Roakl DaliJ _ j 




O iKi: Mj joni U (Flutife Invoci :Ui <“i!]d ill^m ofH I ^vIkisc n:tfVlC v^‘jKS I 

iM] hi,^ lift: hehrid hcxtf^ a Cjiiier rmd |^c<;fn3 peisnn, Hc: w:u very jiorH 

;kad \cjy I i-ipp)‘ 

VX-lien Lahort tliscovcR'iJ rhariie had mke in his hcnise, h did nor hoclu:r him 
muclii at Unit. But die mice nniltlplied. They 0?cgcm to bother him. They kept ojs 
rmihi|>lyinga33ci fioiially thete came a time vviieji he cuyfd m\^d k no longer 
"llii;. h UXi mudU’ lie ^nd *'‘rlii>^ tCci3ly going a bit tuo lari' He hobbletl 
om <»f ilif dovvii ih<^ ro:^d to a simp whci^ he bnight smiiW mcHrseo'^ps. a 
piece i>fchtM!se -^nd ^ome ghiic. 

Wlica he got hume* he put the glue mi the 
u ndeirtiek>di of the mouse tr.ips a nd 5tt:ck them 
to the ceiling. TIicti he baited theiin careluily 
veiti ( pieces of cheese and set iS’jciti io go j>IE 

I hat night wjieti the mice onie out ol tJieir 
Indcs and saw die ctinuscuraps nn the oriling. 

they rkonghr it was a Tremcn-tloits joke. Tlhey 
v^*" 2 lkeiUm>und on the Ifonr. riudgingeadi other 
and pniniiag up with diccr frnjit |mv5 and 
roaring with ilaiiglutr. After all, 5?r was pmey 
siJuV\ mnusarnps nn die cecling. 

When Labort csicrae down she next uiuming 
and saw that div^rsvciC n<i rrth'e ui5?ubt in die 
trap's lir smiled bm nmliing. 

He cook a chair and pm glue rm die biTtoni 
of its L‘[;s and snick k ospside-down to die 
ceiling, near die iaiousetraps. He did die iame 
with the taKe, the retex^scon set and the tamp. 

He li-K‘k evtrydnMg tli.ii %vos<ni the fmiTand 
stuek it cnpvide vkv.vn on the veiling, Hee^’eii piit 
a &itei.ai|ui up ihrie. 
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'Hie itcM night vvlim the mke ca’irc «>uc of tSicir liotc*. they uvre Mill johing 
viiti.1 Liiighing ithouj wh:u they h;:d seen ilie itighi lu-ihie. Bin n<w. when cliey 
hxthcsl u|» ill the ceiling, they stopped liiughiog very suthienlj 

'’tjtHKl gtiicjtnii itier erred iine, "Ixwk up tlieie! 1 liete's the Hoot!" 
"Heavene above!" dioisted iinother "Vtk imisi iK stantKngon diceeiliiig!" 
"I'm hcgifining to feeJ a tittle gi>^hly," eiid soother 
"Alt the blood’s gintig to niy Iktid," xiiil anoihet 
"’niis is tctiilikr siiid a eery senior niouM. with long sshiskei s, "This is 
really teittble! We must do somcihiitg about tt at otKef 

"1 sli.iil taint if 1 base ii» stand on my lieasl aiyv longer!" shouted a 
young mouse. 



"I can't stand it!" 

'’Save Us! Do soniechtng, somelHsdy, s|uick!" 

'Ilicy vvfje gv.iimg liysccrical now, "I himw what we'll do," said the scry 
senior mouk. "We’i'l al 'land on suit liOtds, iheo see II be the tight svay up." 

(hbediemly, they all sttwd on ilieii heads, and aftet a Jong time, one by one 
the)’ faimed Irom a iusEi ol blood to tlieir br.tiits. 

When Liboo came down the nest motning the llooi ssa> liiieted svith mice. 
Quickly he gathered them up and popped them sK in a basket. 

Sit the tliiitg to renteniber is this: svltenevvi' the ssorki seems to be tetiibly 
upsiJ,. do\siii, make suie you keep your teei iirmly on thcgioinui. 
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Appendix C: Readability and Lexile Analyses 



For each passage in NAEP fourth -grade and PIRLS, three 100-word samples were randomly 
chosen and for each 100-word sample, the number of sentences and syllables were counted. 
For one PIRLS passage— a brochure— with fewer than 300 words only two 100-word samples 
were chosen. Sentence and syllable tabuladon was accomplished with the help of a program 
found at: http://www.educational-psychologist.co.uk/fry_readability_program.htm. 

The above-mendoned program also automadcally calculated the "Fry Readability Age." The 
"Average Age" for each passage was calculated by averaging the "Fry Readability Age" from 
the three samples. The "Average Age" for each assessment was calculated by averaging the 
‘‘Average Age'’ from the eight passages of each respecdve assessment. Using the average 
number of sentences and syllables for each passage, the "Average Grade Level" for each 
passage was found using the Fry Graph. To find the “Average Grade Level” for each 
assessment, the Average Grade Level” from the eight passages of each respecdve assessment 
were averaged. 



The same average sentence and syllable counts used in the Fry Analyses were used to 
compute the "Average Reading Ease" for each passage (Appendix, Tables A3 and A4). The 
formula used: ([(Words/Sentences)*1.015] + [(Syllables/Words)*84.6]). The resuldng score is 
on a scale of 1-100, with higher numbers indicadng easier reading levels. To find the 
“Average Reading Ease” for each assessment, the “Average Reading Ease” scores from the 
eight passages of each respecdve assessment were averaged. The qualitadve labels (e.g. 
“Easy,” “Fairly Easy”) attached to these Reading Ease scores were assigned based on 10- 
point intervals, reprinted below. Also assigned based on the table’s transladon of Reading 
Ease scores is the “Average Grade Level”. 



Keadinp Ease Score 
0-29 
30-49 
50-59 
60-69 
70-79 
80-89 
90-100 



Difficulty 
Very Difficult 
Difficult 
Fairly Difficult 
Standard 
Fairly Easy 
Easy 

Very Easy 



Flescb Grade Eevel 
Post Graduate 
College 
High School 
8th to 9th Grade 
7th Grade 
5th to 6th Grade 
4th to 5th Grade 



The Lexile analysis was conducted by using the software provided by MetaMetrics and 
accessed on-line. Each reading passage was formatted according to MetaMetrics guidelines 
and run through the software program to obtain the Lexile score. 
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Appendix D: Classification of Items 

Did the Expert Panel Agree with PIRLS Developers on the Processes Assessed by 
Each PIRLS Item? 

Table D.1: Comparison between Classification by the PIRLS Framework Categories by the PIRLS 
Developers and by the Expert Panel 





Focus on and 
Retrieve Explicitly 
Stated 
Information 


Make 

Straightforward 

Inferences 


Interpret and 
Integrate Ideas 
and Information 


Examine and 
Evaluate Content. 

Language, and 
Textual Elements 


Classified by PIRLS 
Developers 


26% 


28% 


32% 


15% 


Classified by Panel 


24% 


32% 


28% 


16% 



Did the Expert Panel Agree with NAEP Developers on the Processes Assessed by 
Each NAEP Item? 

Table D.2: Comparison between Classification by the NAEP Framework Categories by NAEP 
Developers and by the Expert Panel 





Forming an Initial 
Understanding 


Developing an 
Interpretation 


Personal 
Reflection and 
Response 


Demonstrating 
a Critical 
Stance 


Classified by NAEP 
Developers 


10% 


55% 


15% 


21% 


Classified by Panel 


10% 


72% 


11% 


7% 
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Listing of NCES Working Papers to Date 



Working papers can be downloaded as .pdf files from the NCES Electronic Catalog (http://nces.ed.gov/pubsearch/y 
You can also contact Sheilah Jupiter at (202) 502-7444 (sheilah.jupiter@ed.gov) if you are interested in any of the 
following papers. 



Listing of NCES Working Papers by Program Area 

No. Title NCES contact 



Baccalaureate and Beyond (B&B) 

98-15 Development of a Prototype System for Accessing Linked NCES Data 

2001- 15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 

Methodology Report 

2002- 04 Improving Consistency of Response Categories Across NCES Surveys 



Steven Kaufman 
Andrew G. Malizio 

Marilyn Seastrom 



Beginning 

98-11 

98-15 

1999-15 

2001- 04 

2002- 04 



Postsecondary Students (BPS) Longitudinal Study 

Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

Development of a Prototype System for Accessing Linked NCES Data 

Projected Postsecondary Outcomes of 1992 High School Graduates 

Beginning Postsecondary Students Longitudinal Study: 1996-2001 (BPS: 1996/2001) 

Field Test Methodology Report 

Improving Consistency of Response Categories Across NCES Surveys 



Aurora D’Amico 

Steven Kaufman 
Aurora D’Amico 
Paula Knepper 

Marilyn Seastrom 



Common Core of Data (CCD) 

95- 12 Rural Education Data User’s Guide 

96- 1 9 Assessment and Analysis of School-Level Expenditures 

97- 15 Customer Service Survey: Common Core of Data Coordinators 

97- 43 Measuring Inflation in Public School Costs 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

1999- 03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 

2000- 12 Coverage Evaluation of the 1994-95 Common Core of Data: Public 

Elementary/Secondary School Universe Survey 

2000-13 Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core 
Data (CCD) 

2002-02 School Locale Codes 1 987 - 2000 



of 



Samuel Peng 
William J. Fowler, Jr. 
Lee Hoffman 
William J. Fowler, Jr. 
Steven Kaufman 
Beth Young 

Beth Young 

Kerry Gruber 

Frank Johnson 



Data Development 

2000-1 6a Lifelong Learning NCES Task Force: Final Report Volume I Lisa Hudson 

2000-16b Lifelong Learning NCES Task Force: Final Report Volume II Lisa Hudson 



Decennial Census School District Project 

95- 12 Rural Education Data User’s Guide 

96- 04 Census Mapping Project/School District Data Book 

98-07 Decennial Census School District Project Planning Report 



Samuel Peng 
Tai Phan 
Tai Phan 



Early Childhood Longitudinal Study (ECLS) 

96-08 How Accurate are Teacher Judgments of Students’ Academic Performance? 

96- 18 Assessment of Social Competence, Adaptive Behaviors, and Approaches to Learning with 

Young Children 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies 

97-36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood 

Programs: A Review and Recommendations for Future Research 

1999- 01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

2000- 04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 

1999 AAPOR Meetings 

2001- 02 Measuring Father Involvement in Young Children’s Lives: Recommendations for a 

Fatherhood Module for the ECLS-B 

2001-03 Measures of Socio-Emotional Development in Middle Childhood 



Jerry West 
Jerry West 

Jerry West 
Jerry West 

Jerry West 
Dan Kasprzyk 

Jerry West 

Elvira Hausken 
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BEST 



No. 



Title 



NCES contact 



2001-06 



2002-05 



Papers from the Early Childhood Longitudinal Studies Program: Presented at the 200 1 
A ERA and SRCD Meetings 

Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K), 
Psychometric Report for Kindergarten Through First Grade 



Education Finance Statistics Center (EDFIN) 

94- 05 Cost-of-Education Differentials Across the States 

96- 1 9 Assessment and Analysis of School-Level Expenditures 

97- 43 Measuring Inflation in Public School Costs 

98- 04 Geographic Variations in Public Schools’ Costs 

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model 
Approach 

Education Longitudinal Study: 2002 (ELS:2002) 

2003-03 Education Longitudinal Study: 2002 (ELS: 2002) Field Test Report 

High School and Beyond (HS&B) 

95- 12 Rural Education Data User’s Guide 

1999-05 Procedures Guide for Transcript Studies 
1999-06 1998 Revision of the Secondary School Taxonomy 

2002- 04 Improving Consistency of Response Categories Across NCES Surveys 

HS Transcript Studies 

1999-05 Procedures Guide for Transcript Studies 
1999-06 1998 Revision of the Secondary School Taxonomy 

2003- 01 Mathematics, Foreign Language, and Science Coursetaking and the NELS:88 Transcript 

Data 

2003-02 English Coursetaking and the NELS:88 Transcript Data 



International Adult Literacy Survey (lALS) 

97-33 Adult Literacy: An International Perspective 

Integrated Postsecondary Education Data System (IPEDS) 

97- 27 Pilot Test of IPEDS Finance Survey 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

2000-14 IPEDS Finance Data Comparisons Under the 1997 Financial Accounting Standards for 
Private, Not-for-Profit Institutes: A Concept Paper 

National Assessment of Adult Literacy (NAAL) 

98-17 Developing the National Assessment of Adult Literacy: Recommendations from 
Stakeholders 

1999-09a 1992 National Adult Literacy Survey: An Overview 

1999-09b 1992 National Adult Literacy Survey: Sample Design 

1999-09c 1992 National Adult Literacy Survey: Weighting and Population Estimates 

1999-09d 1992 National Adult Literacy Survey: Development of the Survey Instruments 

1999-09e 1992 National Adult Literacy Survey: Scaling and Proficiency Estimates 

1999-09f 1992 National Adult Literacy Survey: Interpreting the Adult Literacy Scales and Literacy 

Levels 

1999- 09g 1992 National Adult Literacy Survey: Literacy Levels and the Response Probability 

Convention 

2000- 05 Secondary Statistical Modeling With the National Assessment of Adult Literacy: 

Implications for the Design of the Background Questionnaire 
2000-06 Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door 
Surveys in the Assessment of Adult Literacy 

2000-07 “How Much Literacy is Enough?” Issues in Defining and Reporting Performance 
Standards for the National Assessment of Adult Literacy 
2000-08 Evaluation of the 1992 NALS Background Survey Questionnaire: An Analysis of Uses 
with Recommendations for Revisions 

2000- 09 Demographic Changes and Literacy Development in a Decade 

2001- 08 Assessing the Lexile Framework: Results of a Panel Meeting 



Jerry West 



Elvira Hausken 



William J. Fowler, Jr. 
William J. Fowler, Jr. 
William J. Fowler, Jr. 
William J. Fowler, Jr. 
William J. Fowler, Jr. 



Jeffrey Owings 



Samuel Peng 
Dawn Nelson 
Dawn Nelson 
Marilyn Seastrom 



Dawn Nelson 
Dawn Nelson 
Jeffrey Owings 

Jeffrey Owings 



Marilyn Binkley 



Peter Stowe 
Steven Kaufman 
Peter Stowe 



Sheida White 

Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 
Alex Sedlacek 

Alex Sedlacek 

Sheida White 

Sheida White 

Sheida White 

Sheida White 

Sheida White 
Sheida White 
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No. Title 

2002-04 Improving Consistency of Response Categories Across NCES Surveys 

National Assessment of Educational Progress (NAEP) 

95-12 Rural Education Data User’s Guide 

97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 act’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-3 1 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questionnaires) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97- 44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

1999-05 Procedures Guide for Transcript Studies 

1999-06 1998 Revision of the Secondary School Taxonomy 

2001-07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2001-08 Assessing the Lexile Framework: Results of a Panel Meeting 

2001-1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 

2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

2001- 19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 

of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

2002- 04 Improving Consistency of Response Categories Across NCES Surveys 

2002-06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

2002- 07 Teacher Quality, School Context, and Student Race/Ethnicity: Findings from the Eighth 

Grade National Assessment of Educational Progress 2000 Mathematics Assessment 

2003- 10 A Content Comparison of NAEP and PIRLS Fourth-Grade Reading Assessments 



National Education Longitudinal Study of 1988 (N£LS:88) 

95-04 National Education Longitudinal Study of 1988: Second Follow-up Questionnaire Content 
Areas and Research Issues 

National Education Longitudinal Study of 1988: Conducting Trend Analyses ofNLS-72, 
HS&B, and NELS:88 Seniors 

National Education Longitudinal Study of 1988: Conducting Cross-Cohort Comparisons 
Using HS&B, NAEP, and NELS:88 Academic Transcript Data 
National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and 
NELS:88 Sophomore Cohort Dropouts 
Rural Education Data User’s Guide 

Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 
Issues 

National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second 
Follow-Up: Final Methodology Report 

High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 
Development of a Prototype System for Accessing Linked NCES Data 
Procedures Guide for Transcript Studies 
1998 Revision of the Secondary School Taxonomy 
Projected Postsecondary Outcomes of 1992 High School Graduates 
Imputation of Test Scores in the National Education Longitudinal Study of 1988 



95-05 

95-06 

95-07 

95-12 

95- 14 

96- 03 
98-06 
98-09 



98-15 

1999-05 

1999-06 

1999-15 

2001-16 



NCES contact 
Marilyn Seastrom 



Samuel Peng 
Steven Gorman 

Steven Gorman 

Steven Gorman 

Steven Gorman 

Steven Gorman 
Michael Ross 

Steven Kaufman 
Dawn Nelson 
Dawn Nelson 
Arnold Goldstein 



Sheida White 
Arnold Goldstein 
Arnold Goldstein 
Arnold Goldstein 



Marilyn Seastrom 
Arnold Goldstein 



Janis Brown 
Marilyn Binkley 



Jeffrey Owings 

Jeffrey Owings 

Jeffrey Owings 

Jeffrey Owings 

Samuel Peng 
Samuel Peng 

Jeffrey Owings 

Ralph Lee 

Jeffrey Owings 



Steven Kaufriian 
Dawn Nelson 
Dawn Nelson 
Aurora D’Amico 
Ralph Lee 
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BEST 



No. Title 

2002- 04 Improving Consistency of Response Categories Across NCES Surveys 

2003- 01 Mathematics, Foreign Language, and Science Coursetaking and the NELS:88 Transcript 

Data 

2003-02 English Coursetaking and the NELS:88 Transcript Data 

National Household Education Survey (NHES) 

95- 12 Rural Education Data User’s Guide 

96- 13 Estimation of Response Bias in the NHES:95 Adult Education Survey 

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96-21 1993 National Household Education Survey (NHES:93) Questionnaires: Screener, School 

Readiness, and School Safety and Discipline 

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 
96-29 Undercoverage Bias in Estimates of Characteristics of Adults and 0- to 2-Year-Olds in the 
1995 National Household Education Survey (NHES:95) 

96- 30 Comparison of Estimates from the 1995 National Household Education Survey 

(NHES:95) 

97- 02 Telephone Coverage Bias and Recorded Interviews in the 1993 National Household 

Education Survey (NHES:93) 

97-03 1991 and 1995 National Household Education Survey Questionnaires: NHES:9I Screener, 

NHES:91 Adult Education, NHES:95 Basic Screener, and NHES:95 Adult Education 
97-04 Design, Data Collection, Monitoring, Interview Administration Time, and Data Editing in 
the 1993 National Household Education Survey (NHES:93) 

97-05 Unit and Item Response, Weighting, and Imputation Procedures in the 1993 National 
Household Education Survey (NHES:93) 

97-06 Unit and Item Response, Weighting, and Imputation Procedures in the 1995 National 
Household Education Survey (NHES:95) 

97-08 Design, Data Collection, Interview Timing, and Data Editing in the 1995 National 
Household Education Survey 

97-19 National Household Education Survey of 1995: Adult Education Course Coding Manual 

97-20 National Household Education Survey of 1995: Adult Education Course Code Merge 

Files User’s Guide 

97-25 1996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
97-28 Comparison of Estimates in the 1996 National Household Education Survey 
97-34 Comparison of Estimates from the 1993 National Household Education Survey 
97-35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996 
National Household Education Survey 

97-38 Reinterview Results for the Parent and Youth Components of the 1996 National 
Household Education Survey 

97-39 Undercoverage Bias in Estimates of Characteristics of Households and Adults in the 1996 
National Household Education Survey 

97- 40 Unit and Item Response Rates, Weighting, and Imputation Procedures in the 1996 

National Household Education Survey 

98- 03 Adult Education in the 1990s: A Report on the 1991 National Household Education 

Survey 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

2002-04 Improving Consistency of Response Categories Across NCES Surveys 

National Longitudinal Study of the High School Class of 1972 (NLS'72) 

95- 12 Rural Education Data User’s Guide 

2002-04 Improving Consistency of Response Categories Across NCES Surveys 

National Postsecondary Student Aid Study (NPSAS) 

96- 17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 

2000-17 National Postsecondary Student Aid Study:2000 Field Test Methodology Report 




NCES contact 
Marilyn Seastrom 
Jeffrey Owings 

Jeffrey Owings 

Samuel Peng 
Steven Kaufman 
Steven Kaufman 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 
Peter Stowe 

Kathryn Chandler 

Kathryn Chandler 
Kathryn Chandler 
Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 

Peter Stowe 

Marilyn Seastrom 

Samuel Peng 
Marilyn Seastrom 

Andrew G. Malizio 
Andrew G. Malizio 
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No. Title 

2002-03 National Postsecondary Student Aid Study, 1999-2000 (NPSAS:2000), CATl 
Nonresponse Bias Analysis Report. 

2002-04 Improving Consistency of Response Categories Across NCES Surveys 

National Study of Postsecondary Faculty (NSOPF) 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2002-04 Improving Consistency of Response Categories Across NCES Surveys 
2002-08 A Profile of Part-time Faculty: Fall 1998 

Postsecondary Education Descriptive Analysis Reports (PEDAR) 

2000-1 1 Financial Aid Profile of Graduate Students in Science and Engineering 



Private School Universe Survey (PSS) 

95-16 Intersurvey Consistency in NCES Private School Surveys 

95- 17 Estimates of Expenditures for Private K-12 Schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

96-26 Improving the Coverage of Private Elementary-Secondary Schools 

96- 27 Intersurvey Consistency in NCES Private School Surveys for 1993-94 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97- 22 Collection of Private School Finance Data: Development of a Questionnaire 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 

2000-15 Feasibility Report: School-Level Finance Pretest, Private School Questionnaire 
Recent College Graduates (RCG) 

98-15 Development of a Prototype System for Accessing Linked NCES Data 
2002-04 Improving Consistency of Response Categories Across NCES Surveys 



Schools and Staffing Survey (SASS) 

94-01 Schools and Staffing Survey (SASS) Papers Presented at Meetings of the American 
Statistical Association 

94-02 Generalized Variance Estimate for Schools and Staffing Survey (SASS) 

94-03 1991 Schools and Staffing Survey (SASS) Reinterview Response Variance Report 

94-04 The Accuracy of Teachers* Self-reports on their Postsecondary Education: Teacher 

Transcript Study, Schools and Staffing Survey 

94- 06 Six Papers on Teachers from the 1990-91 Schools and Staffing Survey and Other Related 

Surveys 

95- 01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American 

Statistical Association 

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing 
QED School Estimates with CCD Estimates 
95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis 
95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates 
95-09 The Results of the 1993 Teacher List Validation Study (TLVS) 

95-10 The Results of the 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive 



Reconciliation 

95—1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

95-12 Rural Education Data User’s Guide 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

95-15 Classroom Instructional Processes: A Review of Existing Measurement Approaches and 
Their Applicability for the Teacher Follow-up Survey 
95-16 Intersurvey Consistency in NCES Private School Surveys 

95-18 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and 

Staffing Survey 




NCES contact 
Andrew Malizio 

Marilyn Seastrom 



Linda Zimbler 
Steven Kaufman 
Linda Zimbler 
Marilyn Seastrom 
Linda Zimbler 



Aurora D’Amico 



Steven Kaufman 
Stephen Broughman 
Stephen Broughman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 

Stephen Broughman 
Steven Kaufman 
Dan Kasprzyk 

Stephen Broughman 



Steven Kaufman 
Marilyn Seastrom 



Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Sharon Bobbitt & 
John Ralph 
Samuel Peng 
Samuel Peng 

Sharon Bobbitt 

Steven Kaufman 
Dan Kasprzyk 
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Title 



NCES contact 



No. 

96-01 

96-02 

96-05 

96-06 

96-07 

96-09 

96-10 

96-11 

96-12 

96-15 

96-23 

96-24 

96-25 

96- 28 

97- 01 

97-07 

97-09 

97-10 

97-11 

97-12 

97-14 

97-18 

97-22 

97-23 

97-41 

97-42 

97- 44 

98- 01 
98-02 
98-04 
98-05 

98-08 

98-12 

98-13 

98-14 

98-15 

98-16 

1999-02 

1999-04 

1999-07 

1999-08 

1999-10 



Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly 
Longitudinal Study 

Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting 
of the American Statistical Association 

Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey 
The Schools and Staffing Survey (SASS) for 1998-99: Design Recommendations to 
Inform Broad Education Policy 

Should SASS Measure Instructional Processes and Teacher Effectiveness? 

Making Data Relevant for Policy Discussions: Redesigning the School Administrator 
Questionnaire for the 1998-99 SASS 
1998-99 Schools and Staffing Survey: Issues Related to Survey Depth 
Towards an Organizational Database on America’s Schools: A Proposal for the Future of 
SASS, with comments on School Reform, Governance, and Finance 
Predictors of Retention, Transfer, and Attrition of Special and General Education 
Teachers: Data from the 1 989 Teacher Followup Survey 
Nested Structures: District-Level Data in the Schools and Staffing Survey 
Linking Student Data to SASS: Why, When, How 
National Assessments of Teacher Quality 

Measures of Inservice Professional Development: Suggested Items for the 1998-1999 
Schools and Staffing Survey 

Student Learning, Teaching Quality, and Professional Development: Theoretical 

Linkages, Current Measurement, and Recommendations for Future Data Collection 
Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the 
American Statistical Association 

The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 
Status of Data on Crime and Violence in Schools: Final Report 

Report of Cognitive Research on the Public and Private School Teacher Questionnaires 
for the Schools and Staffing Survey 1993-94 School Year 
International Comparisons of Inservice Professional Development 
Measuring School Reform: Recommendations for Future SASS Data Collection 
Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and 
Analysis 

Improving the Mail Return Rates of SASS Surveys: A Review of the Literature 
Collection of Private School Finance Data: Development of a Questionnaire 
Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing 
Form 

Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1 997 Meeting 
of the American Statistical Association 

Improving the Measurement of Staffing Resources at the School Level: The Development 
of Recommendations for NCES for the Schools and Staffing Survey (SASS) 
Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 
State Assessments and State NAEP, Feasibility Study 
Collection of Public School Expenditure Data: Development of a Questionnaire 
Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report 
Geographic Variations in Public Schools’ Costs 

SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for 

Determining the Numerators for the SASS Private School (3B) Second-Stage Factors 
The Redesign of the Schools and Staffing Survey for 1 999-2000: A Position Paper 
A Bootstrap Variance Estimator for Systematic PPS Sampling 
Response Variance in the 1994-95 Teacher Follow-up Survey 
Variance Estimation of Imputed Survey Data 

Development of a Prototype System for Accessing Linked NCES Data 
A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 
Measuring Teacher Qualifications 

Collection of Resource and Expenditure Data on the Schools and Staffing Survey 
Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest 
Results to Improve Item Construction 
What Users Say About Schools and Staffing Survey Publications 



Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Mary Rollefson 

Dan Kasprzyk 

Stephen Broughman 

Lee Hoffman 
Dan Kasprzyk 

Dan Kasprzyk 
Mary Rollefson 
Steven Kaufman 

Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 

Steve Kaufman 

Mary Rollefson 

Michael Ross 

Stephen Broughman 
Steven Kaufman 
William J. Fowler, Jr. 
Steven Kaufman 

Dan Kasprzyk 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 
Dan Kasprzyk 
Stephen Broughman 
Dan Kasprzyk 

Dan Kasprzyk 
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5IST COPY 



No. 

1999-12 

1999-13 

1999-14 

1999- 17 

2000- 04 

2000-10 

2000-13 

2000-18 

2002-04 



Title 

1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use 
Codebook 

1993- 94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 
Indian Affairs (BIA) Restricted-Use Codebook 

1994- 95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 
Secondary Use of the Schools and Staffing Survey Data 

Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 

A Research Agenda for the 1 999-2000 Schools and Staffing Survey 
Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 
Data (CCD) 

Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire 
Improving Consistency of Response Categories Across NCES Surveys 



NCES contact 
Kerry Gruber 

Kerry Gruber 

Kerry Gruber 
Susan Wiley 
Dan Kasprzyk 

Dan Kasprzyk 
Kerry Gruber 

Stephen Broughman 
Marilyn Seastrom 



Third International Mathematics and Science Study (TIMSS) 

2001-01 Cross-National Variation in Educational Preparation for Adulthood: From Early 
Adolescence to Young Adulthood 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

2001- 07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2002- 01 Legal and Ethical Issues in the Use of Video in Education Research 



Elvira Hausken 

Patrick Gonzales 
Arnold Goldstein 

Patrick Gonzales 



Listing of NCES Working Papers by Subject 



No. Title 



NCES contact 



Achievement (student) - mathematics 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 



Adult education 

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 

98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education 
Survey 

98-10 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

1999-1 1 Data Sources on Lifelong Learning Available from the National Center for Education 



Statistics 

2000-1 6a Lifelong Learning NCES Task Force: Final Report Volume I 

2000-1 6b Lifelong Learning NCES Task Force: Final Report Volume II 



Adult literacy — see Literacy of adults 



American Indian - education 

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 

Indian Affairs (BIA) Restricted-Use Codebook 



Assessment/achievement 
95-12 Rural Education Data User’s Guide 

95-13 Assessing Students with Disabilities and Limited English Proficiency 
97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 act’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-31 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questions) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97 - 44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

2001-07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2001-1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 
2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

2001- 19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 

of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

2002- 05 Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K), 

Psychometric Report for Kindergarten Through First Grade 



Patrick Gonzales 

Steven Kaufman 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 

Peter Stowe 

Lisa Hudson 

Lisa Hudson 
Lisa Hudson 



Kerry Gruber 



Samuel Peng 
James Houser 
Larry Ogle 
Larry Ogle 

Larry Ogle 

Larry Ogle 

Larry Ogle 
Michael Ross 

Jeffrey Owings 



Arnold Goldstein 



Arnold Goldstein 
Arnold Goldstein 
Arnold Goldstein 



Elvira Hausken 
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lEST COPY AVAIMBLIE 



No. Title 

2002-06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

2002- 07 Teacher Quality, School Context, and Student Race/Ethnicity: Findings from the Eighth 

Grade National Assessment of Educational Progress 2000 Mathematics Assessment 

2003- 10 A Content Comparison of the NAEP and PIRLS Fourth-Grade Reading Assessments 

Beginning students in postsecondary education 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

2001-04 Beginning Postsecondary Students Longitudinal Study: 1996-2001 (BPS: 1996/2001) 

Field Test Methodology Report 

Civic participation 

97-25 1996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Climate of schools 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

Cost of education indices 

94-05 Cost-of-Education Differentials Across the States 



Course-taking 



95-12 

98-09 



1999-05 

1999-06 

2003-01 

2003-02 



Rural Education Data User’s Guide 

High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 
Procedures Guide for Transcript Studies 
1998 Revision of the Secondary School Taxonomy 

Mathematics, Foreign Language, and Science Coursetaking and the NELS:88 Transcript 
Data 

English Coursetaking and the NELS:88 Transcript Data 



Crime 

97-09 



Status of Data on Crime and Violence in Schools: Final Report 



Curriculum 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Customer service 

1999- 10 What Users Say About Schools and Staffing Survey Publications 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 

1999 AAPOR Meetings 



Data quality 

97-13 Improving Data Quality in NCES: Database-to-Report Process 
2001-1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 
2001-1 3 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

2001-19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 
of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 



NCES contact 
Arnold Goldstein 



Janis Brown 
Marilyn Binkley 



Aurora D’Amico 
Paula Knepper 



Kathryn Chandler 



Samuel Peng 



William J. Fowler, Jr. 



Samuel Peng 
Jeffrey Owings 



Dawn Nelson 
Dawn Nelson 
Jeffrey Owings 

Jeffrey Owings 



Lee Hoffman 



Sharon Bobbitt & 
John Ralph 
Jeffrey Owings 



Dan Kasprzyk 
Valena Plisko 
Dan Kasprzyk 



Susan Ahmed 
Arnold Goldstein 
Arnold Goldstein 
Arnold Goldstein 



No. Title 

2002-06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 



Data warehouse 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 



Design effects 

2000- 03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 

Variances from NCES Data Sets 

Dropout rates, high school 

95- 07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and 

NELS:88 Sophomore Cohort Dropouts 

Early childhood education 

96- 20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies 

97- 36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood 

Programs: A Review and Recommendations for Future Research 

1999- 01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

2001- 02 Measuring Father Involvement in Young Children's Lives: Recommendations for a 

Fatherhood Module for the ECLS-B 

2001-03 Measures of Socio-Emotional Development in Middle School 

2001- 06 Papers from the Early Childhood Longitudinal Studies Program: Presented at the 2001 

AERA and SRCD Meetings 

2002- 05 Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K), 

Psychometric Report for Kindergarten Through First Grade 

Educational attainment 

98- 1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 

Test Report 

2001- 15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 

Methodology Report 

Educational research 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2002- 01 Legal and Ethical Issues in the Use of Video in Education Research 

Eighth-graders 

2001- 05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

2002- 07 Teacher Quality, School Context, and Student Race/Ethnicity: Findings from the Eighth 

Grade National Assessment of Educational Progress 2000 Mathematics Assessment 



Employment 

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 
Issues 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

2000-1 6a Lifelong Learning NCES Task Force: Final Report Volume I 

2000- 1 6b Lifelong Learning NCES Task Force: Final Report Volume II 

2001- 01 Cross-National Variation in Educational Preparation for Adulthood: From Early 

Adolescence to Young Adulthood 

Employment - after college 



NCES contact 
Arnold Goldstein 



Dan Kasprzyk 



Ralph Lee 



Jeffrey Owings 



Kathryn Chandler 

Kathryn Chandler 

Jerry West 
Jerry West 

Jerry West 
Jerry West 

Elvira Hausken 
Jerry West 



Elvira Hausken 



Aurora D’Amico 
Andrew G. Malizio 



Valena Plisko 
Patrick Gonzales 



Patrick Gonzales 
Janis Brown 



Jeffrey Owings 

Aurora D’Amico 

Lisa Hudson 
Lisa Hudson 
Elvira Hausken 
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No. Title 

2001-15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 
Methodology Report 



Engineering 

2000-1 1 Financial Aid Profile of Graduate Students in Science and Engineering 



Enrollment - after college 

2001- 15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 

Methodology Report 

Faculty - higher education 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2002- 08 A Profile of Part-time Faculty: Fall 1998 

Fathers - role in education 

2001- 02 Measuring Father Involvement in Young Children's Lives: Recommendations for a 

Fatherhood Module for the ECLS-B 

Finance - elementary and secondary schools 
94-05 Cost-of-Education Differentials Across the States 

96- 19 Assessment and Analysis of School-Level Expenditures 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1999- 16 Measuring Resources in Education: From Accounting to the Resource Cost Model 

Approach 

2000- 18 Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire 

Finance - postsecondary 

97- 27 Pilot Test of IPEDS Finance Survey 

2000-14 IPEDS Finance Data Comparisons Under the 1997 Financial Accounting Standards for 
Private, Not-for-Profit Institutes: A Concept Paper 



Finance - 

95- 17 

96- 16 

97- 07 

97-22 

1999- 07 

2000- 15 



private schools 

Estimates of Expenditures for Private K-12 Schools 
Strategies for Collecting Finance Data from Private Schools 
The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
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Public schools 

97- 43 Measuring Inflation in Public School Costs 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 
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98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 
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Processing, and Editing Cycle 

School districts, public - demographics of 
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Schools 

97- 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations forNCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 
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Schools - safety and discipline 

97-09 Status of Data on Crime and Violence in Schools: Final Report 
Science 

2000- 1 1 Financial Aid Profile of Graduate Students in Science and Engineering 

2001- 07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

Software evaluation 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 

Staff 

97— 42 Improving the Measurement of Staffing Resources at the School Level: The Development 

of Recommendations for NCES for the Schools and Staffing Survey (SASS) 

98- 08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 

Staff - higher education institutions 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2002- 08 A Profile of Part-time Faculty: Fall 1998 

Staff - non professional 

2000- 13 Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 

Data (CCD) 

State 

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 
Processing, and Editing Cycle 

Statistical methodology 

97-21 Statistics for Policymakers or Everything You Wanted to Know About Statistics But 
Thought You Could Never Understand 

Statistical standards and methodology 

2001- 05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 

2002- 04 Improving Consistency of Response Categories Across NCES Surveys 

Students with disabilities 

95- 13 Assessing Students with Disabilities and Limited English Proficiency 

2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

Survey methodology 

96- 17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 

97- 15 Customer Service Survey: Common Core of Data Coordinators 

97- 35 Design, Data Collection, Interview Administration Time, and Data Editing in the 1996 

National Household Education Survey 

98- 06 National Education Longitudinal Study of 1988 (NELS:88) Base Year through Second 

Follow-Up: Final Methodology Report 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

98-16 A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1999- 17 Secondary Use of the Schools and Staffing Survey Data 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2000-02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 
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Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
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98-13 Response Variance in the 1994-95 Teacher Follow-up Survey 

1999— 14 1994-95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 

2000- 10 A Research Agenda for the 1999-2000 Schools and Staffing Survey 

2002-07 Teacher Quality, School Context, and Student Race/Ethnicity: Findings from the Eighth 
Grade National Assessment of Educational Progress 2000 Mathematics Assessment 

Teachers - instructional practices of 

98-08 The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 
2002-06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

Teachers - opinions regarding safety 
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1999-04 Measuring Teacher Qualifications 
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94-05 Cost-of-Education Differentials Across the States 
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